Least Squares Regression

• Regression, generally, is choosing a function of covariates \(X_i\) to predict outcomes \(Y_i\).
• We choose from a set of functions \(\mathcal{M}\) that we call a regression model.

\[ Y_i \approx \hat\mu(X_i) \qfor \hat\mu \in \mathcal{M} \]

• Least Squares Regression is a special case in which we choose by minimizing the sum of squared residuals.

\[ \hat \mu = \mathop{\mathrm{argmin}}_{m \in \mathcal{M}} \sum_{i=1}^n \qty{ Y_i - m(X_i) }^2 \]

• Given a model, we can find the least squares estimator by doing this minimization.
• And we can do this for any model we like, so long as we can solve the minimization problem. \[ \begin{aligned} \textcolor{red}{\mathcal{M}} &= \qty{ \text{all functions} \ m(x) } \\ \textcolor{blue}{\mathcal{M}} &= \qty{ \text{all lines} \ m(x)=ax + b } \\ \textcolor{magenta}{\mathcal{M}} &= \qty{ \text{all increasing functions} \ m(x) } \\ \end{aligned} \]

Linear Models

• In this class, we’re going to focus on linear models.
  • These are sets of functions that are ‘closed’ under addition, subtraction, and scaling by constants.
  • This means when we add, subtract, and scale functions in the model, we get another function in the model. \[ a(x), b(x) \in \mathcal{M}\implies a(x) + b(x), a(x) - b(x), c a(x) \in \mathcal{M}\quad \text{ for any constant $c$ } \]
• Q. Which of these models are linear?

\[ \begin{aligned} \textcolor{red}{\mathcal{M}} &= \qty{ \text{all functions} \ m(x) } \\ \textcolor{blue}{\mathcal{M}} &= \qty{ \text{all lines} \ m(x)=ax + b } \\ \textcolor{magenta}{\mathcal{M}} &= \qty{ \text{all increasing functions} \ m(x) } \\ \end{aligned} \]

The Additive Bivariate Model

• Here we let our predictions be arbitrary functions of education (\(x\)) as before.
• But we’re requiring that race (\(w\)) only shift that function up and down—it can’t change the function’s shape.
• The result, in visual terms, are predictions for different values of \(w\) are ‘parallel curves’.

\[\small{ \begin{aligned} \textcolor{blue}{\mathcal{M}} &= \qty{ m(w,x) = m_0(w) + m_1(x) \ \ \text{for univariate functions} \ \ m_0, m_1 } &&\text{additive bivariate model} \\ \end{aligned} } \]

The Parallel Lines Model

• Here we impose the additional restriction that the function of education (\(x\)) is a line.
• This means that the predictions for different values of \(w\) are parallel lines.

\[\small{ \begin{aligned} \textcolor{red}{\mathcal{M}} &= \qty{ m(w,x) = a(w) + b x } &&\text{parallel lines} \\ \end{aligned} } \]

The Not-Necessarily-Parallel Lines Model

• Here we stick with the lines thing, but drop the additivity restriction.
• For each value of \(w\), we can get a totally different line.

\[\small{ \begin{aligned} \textcolor{magenta}{\mathcal{M}} &= \qty{ m(w,x) = a(w) + b(w) x } &&\text{lines} \\ \end{aligned} } \]

Where This Leaves Us

• Each of these approaches allows us to make predictions everywhere—even in columns we haven’t sampled.
• Fundamentally, it’s because they’re using information from one column to make predictions in another.
• This solves the empty-groups problem, but it introduces a new one. What is it?

What It Is

• Bias is the difference between an estimator’s expected value and what we mean to estimate.
• Intuitively, it’s the systematic part of our estimator’s error.
• Visually, it’s the difference between …
  • The location of our estimator’s sampling distribution. The black line.
  • The location of what we’re trying to estimate. The green line.

Q. What is bias here?

Bias and Consistency

• In Practice Midterm 1, we looked at the sampling distribution of three estimators at three sample sizes.
  • Estimator a was unbiased.
  • Estimator b was biased, but its bias decreased with sample size.
  • Estimator c was biased and its bias didn’t change with sample size.
• Constant bias means our estimator isn’t going to much closer to the truth as we get more data.
• We call estimators like this—estimators that don’t get arbitrarily close to the truth in large samples—inconsistent.

A Biased Least Squares Estimator

\[ \begin{aligned} \textcolor{red}{\mathcal{M}} &= \qty{ \text{all functions} \ m(x) } \\ \textcolor{blue}{\mathcal{M}} &= \qty{ \text{all lines} \ m(x)=ax + b } \\ \end{aligned} \]

• We can see that, when we use the all lines model, our least squares estimate \(\hat\mu(8)\) is biased.
• Why? Because the best line, trying to get the slope right in the middle where most of the data is, gets it wrong at the edges.
• Typically, we can’t know an estimator is biased, but we can suspect it is.
  • e.g., because it’s nowhere near the subsample means.
• That’s because bias is a property of the estimator’s sampling distribution. And we can’t observe that directly.
• But in this case something gives it away: \(\hat\mu(8)\) is a negative estimate of a positive quantity.

Bias and Fake Data

• To really see bias, we have to work with fake data.
• We’ll work with a fake version of the earnings data we’ve been using.
  • Earlier in the semester, we’ve used a fake version of our income data for illustration.
  • Now we’re actually going to sample from it.

fake.sample = fake.pop[sample(1:nrow(fake.pop), size=n, replace=TRUE), ]

• And we can actually calculate our estimator’s error.

error = mean(fake.pop$income[fake.pop$education==8]) - muhat(8)

• And therefore its bias, the mean of our estimator’s error over many samples.

Bias and Fake Data

• We can see, looking at our estimator’s sampling distribution, that it has downward (i.e. negative) bias.

muhat8.sampling.distribution = replicate(1000, {
  fake.sample = fake.pop[sample(1:nrow(fake.pop), size=n, replace=TRUE), ]
  model = lm(income~education, data=fake.sample)
  predict(model, newdata=data.frame(education=8))
})

Q. Does this go away with more data?

What This Is About

• Bias doesn’t go away with more data. What we’re seeing here is …
  • In blue, our model’s fit to the population.
  • In cyan, its fit to a sample of size \(n/4\).
  • In magenta, its fit to a sample of size \(n\).
  • In orange, its fit to a sample of size \(4n\).
• As sample size increases, we get closer to the blue line.
• But the blue line isn’t close to the subpopulation (i.e. column) means \(\textcolor{red}{\hat\mu(x)}\).

Least Squares Bias

• This isn’t great, but at least we know what’s going on.
• In fact, we can prove that the bias of the least squares estimator \(\hat\mu(8)\) …
  • is the error we get when we fit the model to the whole population
  • no matter what sample size we’re using
• As sample size increases …
  • our estimator’s sampling distribution gets narrower and narrower
  • but—if we have bias—it get narrower around the wrong value.

It’s Not Just About Lines

• When we use the additive bivariate model to predict income from education and race …
  • some subpopulation means are still far from the model’s predictions
  • i.e. some predictions \(\textcolor[RGB]{0,0,255}{\hat\mu(x)}\) are still biased estimators of \(\textcolor[RGB]{248,118,109}{\mu(x)}\)

But Lines Aren’t Great

• That said, when we use the additive lines model to predict income from education and race, we tend to do worse.
  • The misspecification, or ‘wrongness’, of a model is a matter of degree.
  • And this one is more wrong.

Two Versions of Our MHT Experiment

• Researchers hypothesize that MHT increases bone density. We want to know is how well it works, on average, for the people who take it.
• Above, we’ve got a population of 10K people eligible for some form of MHT randomized two ways.
  • Their age at the time the study begins is on the \(x\)-axis.
  • Bone density, measured 2 years later, is on the \(y\)-axis.
  • Color indicates whether they did or didn’t do MHT.
  • On the left, randomization probabilities depend on age. On the right, they don’t.
• How does the way we randomize affect the bias of our estimator when we use a bad model?

Version 1: Age-Dependent Randomization

\[ \begin{aligned} \textcolor[RGB]{0,191,196}{\frac{1}{m_1}\sum_{j:w_j=1}} \qty{ \textcolor[RGB]{0,191,196}{\hat\mu(1,x_j)} - \textcolor[RGB]{248,118,109}{\hat\mu(0,x_j)} } &= 0.667 - 0.501 = 0.166 && \text{all functions} \\ &= 0.683 - 0.56 = 0.122 && \text{additive} \\ &= 0.68 - 0.564 = 0.116 && \text{lines} \\ \end{aligned} \]

Version 2: Age-Independent Randomization

\[ \begin{aligned} \textcolor[RGB]{0,191,196}{\frac{1}{m_1}\sum_{j:w_j=1}} \qty{ \textcolor[RGB]{0,191,196}{\hat\mu(1,x_j)} - \textcolor[RGB]{248,118,109}{\hat\mu(0,x_j)} } &= 0.665 - 0.504 = 0.162 && \text{all functions} \\ &= 0.665 - 0.503 = 0.162 && \text{additive} \\ &= 0.665 - 0.503 = 0.162 && \text{lines} \\ \end{aligned} \]